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In this paper, we report on developments in the Mastery Learning (ML) curriculum and 
assessment model that has been successfully implemented in a metropolitan university for 
teaching first-year mathematics. Initial responses to ML were positive; however, we ask 
whether the nature of the ML tests encourages a focus on shallow learning of procedures, 
and whether the structure of the assessment regime provides sufficient motivation for 
learning more complex problem solving. We analysed assessment data, as well as student 
reports and survey responses in an attempt to answer these questions. 


Mastery Learning (ML) has been defined as “both a philosophy of instruction and a set 
of methods for teaching and learning” (Groen, Coupland, Memar, & Langtry, 2016, p. 69). 
Guskey (2010) attributes the development of ML to Bloom (1971). Fundamental to 
Bloom’s (1971) ideas is the belief that most students “‘can learn a subject to a high level of 
mastery... if given sufficient learning time and appropriate types of help” (p. 51). 

Typically, ML involves the identification within each topic of essential skills and 
concepts to be learned, and the use of formative tests, as each topic has been studied, either 
individually as part of a personalised system of instruction (PSI), where students may 
proceed at different rates, or in teacher-led classes in learning for mastery (LFM), where 
the class proceeds together. A high standard (70% to 80%) on tests is set as the level 
necessary to demonstrate “mastery” of each topic. Students who do not reach the mastery 
level on their first attempt at a mastery test are given extra instruction opportunities and 
may re-sit the test several times. This approach has been reported as a successful strategy 
in meeting the challenges faced by university teachers whose students arrive with a range 
of mathematical preparedness, with many lacking the assumed or recommended prior 
knowledge for their university studies (Bradley, 2016; Groen, Coupland, Langtry, Memar, 
Moore, & Stanley, 2015). 

A major review of ML was undertaken by Kulik, Kulik, and Bangert-Drowns (1990) 
who used a meta-analytic methodology to integrate findings from 108 studies of ML 
initiatives at school and university levels. They concluded that ML approaches resulted in 
significant positive effects on student learning, as measured by exam performance given at 
the end of instruction. Where student attitudes to instructional method and to the subject 
were evaluated, those were also mostly positive. It should be noted that only 15 of the 108 
studies were concerned with “college” mathematics. The use of ML and PSI within “Math 
Emporium” re-designs for developmental mathematics courses for underprepared students 
at universities in the U.S. has been lauded as a great step forward (Twigg, 2011). In a 
“Math Emporium”, students are required to attend a minimum number of hours in a 
computer lab using online instructional and testing software. Tutors are on hand to answer 
questions and to provide guidance. This model is not without its critics, who point to 
problems in depth of learning (Almy, 2012) and deleterious effects on those who struggle 
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with individualised instruction and find themselves isolated and unsuccessful (Cordes, 
2014). 

The adaptation of ML used in our institution for first-year mathematics classes is 
described in Groen et al. (2015) and Groen et al. (2016). The model is LFM with 
traditional lectures and tutorials, not a “Math Emporium”. The implementation has evolved 
over time in response to contextual constraints and to the different priorities of subject co- 
ordinators and the skills of the teaching staff. While we are committed to ML as a 
philosophy, and are pleased with the improved pass rates, we realise that all assessment 
regimes can encourage responses from students that include undesirable behavioural 
choices. This is often not foreseen when new assessment policies and practices are 
constructed, and adjustments are required. In this paper, we describe a pilot study and point 
the way to evidence-based adjustments with the goal of improving student learning. A brief 
outline of the way ML is currently implemented into two of our largest subjects is 
provided, followed by a discussion of the concerns that have arisen, the analysis of data 
undertaken to find evidence for the strength of those concerns, and finally the innovations 
that we are planning in order to address the concerns. 


Mastery Learning in our Subjects 


In this paper, we report on ML in two first-year mathematics subjects that are taken by 
all first-year Engineering students at our institution. The first subject, Mathematical 
Modelling 1 (MM1), includes complex numbers, vectors and 3D geometry, calculus, first- 
and second-order differential equations, and modelling using these concepts. The second 
subject, Mathematical Modelling 2 (MM2), builds upon MM1 and includes functions and 
calculus of functions of several variables, partial derivatives, optimisation, multiple 
integrals, and statistics, with an introduction to inference and linear regression. The weekly 
learning activities of the first subject consist of two one-hour lectures, a one-hour tutorial, 
and a one-hour computing lab. Eight of the ten computing labs are allocated for mastery 
tests. The format of the second subject differs in that it has two 1.5-hour-long lectures, one 
on mathematics and one on statistics. The tutorials and computer labs are similar. The 
subjects are usually taken in consecutive semesters. First, we report on the cohort of 418 
students who completed the first subject in first semester, Autumn 2016, and the follow-on 
subject in the next semester, Spring 2016. Second, we report on the cohort of 76 students 
who completed MM1 in Spring 2015 and MM2 in Autumn 2016. 

As described in Groen et al. (2015, 2016), ML was introduced in 2014 to improve 
student learning by ensuring that students mastered the basics of these subjects, which in 
turn led to improved pass rates. Other positive outcomes included lower examination stress 
as the final examinations were made optional. Students required at least 80% on each of 
four mastery tests during the semester, with three opportunities given for students to 
achieve this level. The tests were taken online under supervision in computer labs. 
Students were able to see their results for each test as soon as they submitted their answers. 
Students who achieved 80% to 100% on the mastery tests were allocated marks of 50% to 
62.5% on a linear scale towards their final mark for the subject. To earn more than 62.5% 
and qualify for a Credit, Distinction, or High Distinction, students sat the optional final 
examination, which tested more complex problem-solving skills. 

During 2016, concerns were raised by members of the teaching team that some 
students were reporting that they found the online mastery tests to be too easy and that 
preparation to pass the tests did not prepare students for the final examination. There was 
also a concern that the nature of the tests encouraged surface learning and pattern 


matching. It was noticed in MM1 that attendance at tutorials dropped as the semester 
progressed, with low attendance towards the end of semester, when the emphasis was on 
preparing students for the final examination. In MM2, tutorial attendance dropped 
significantly after the first few weeks, remaining low for the rest of semester. Participation 
in the optional final examination in MM2 was lower than anticipated, based on the 
participation in MM1. To investigate the situation, a plan was put in place to collect and 
analyse relevant data. 


Research Questions, Data Collection, and Analysis 


After discussion with the subject co-ordinators of the relevant subjects, the following 
were formed as our research questions: 
1. Is there any evidence that the students’ experiences in the ML assessment 
contribute to a lack of interest and effort in the final examination? 
2. What do students say about the perceived level of difficulty in the mastery tests? 


The data available for us to investigate these questions consist of: 
¢ All student assessment marks in relevant subjects 
¢ Subject Feedback Survey (SFS) data for relevant subjects (Likert scale 
questions and open response questions) 
¢ An opt-in survey for MM1 Autumn 2016 (76 students responded, of whom 65 
took MM1 in Autumn semester 2016 and MM2 in Spring semester 2016) 


Analysis for Students in MM1 Autumn 2016 


For the sequence of subjects MM1 in Autumn followed by MM2 in Spring semester, 
Table 1 indicates the participation levels in the optional final examinations. 


Table 1 
Final Examination Participation of Students Who Attempted MM1 and MM2 Consecutively 
in the Autumn and Spring Semesters of 2016 


Did not sit MM2 exam Sat MM2 exam Total 
Did not sit MM1 exam 36 8 44 
Sat MM1 exam 79 299 B12 
Totals 115 303 418 


From the data in Table 1, we see that 372/418 or 89% of these students sat the final 
examination in MM1, but only 303/418 or 72% chose to sit the examination in MM2. 
Given that the assessment regime requires students to sit for the final examination in order 
to access grades higher than a Pass, this is a concerning trend. A possible interpretation is 
that students in the second subject are less likely to target their efforts towards achieving 
higher grades. Inspection of the student comments in the SFS for MM2 did not shed any 
light on this issue. 

The data shown in Figure 1 are the distributions of marks in the mastery tests (MT) and 
the final examinations for students who attempted both examinations. For these students, 
the distributions reflect some of the differences in the way assessment was organised in 
each subject. For MM1, students were allowed a second attempt at each individual 
question in the mastery tests in case they had made a typing error. Many students were able 


to achieve 100% scores on the mastery tests. This was not the case for MM2. In MMI, 
students were allowed an A4 sheet of self-prepared notes in the final examination, but in 
MM2, a formula sheet was provided. Overall, the marks obtained in MM2 were lower than 
in MM1, and the examination marks in MM2 were skewed to the right. 
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Figure 1. Performance in mastery tests (MT) and in final examinations (N = 295). 


To further investigate student performance on the final examinations, scatterplots were 
prepared, which compared mastery test (MT) scores with final examination scores for the 
295 students who sat for both examinations in these consecutive semesters (see Figure 2). 


MM 1 Exam Score (out of 100) vs MT Score (out of 62.5) A2016 MM2 Exam Score (out of 100) vs MT Score (out of 62.5) S2016 
re Grade ‘0 Qf Grade 
90 a MM1 a MM2 
4 e oP ik *- *® : e Pp 
80 re soc a * hk aes 
7 7 8 4 a A xe a ™ a 4 4] /% 4 
© © rvN A 
° Bd S o ° ° 
30 o r oe *s 3 60 ° a o o,? r3 t 
= 50 ¢ oo 3 + a rT] . a ra ei % ? o 
5 | ons %, = ak =n + + % 
— 4 aon 7 = g@ — 40 i: oo” age ° 
5 La oe Le 5 oa " al = 
30 : + 7 il oon fs « « a4 es) * 
tee . re ee eT 
s a s a 
20 a? . * otis wa 20 On oF | oor 8 TERS vu " aja 7 
0 6 . s r ah" o eee eo ate ae? te «8° ison 
id e® L | cae le s ee ous we 8 gee tae ee 
0 0 
50 52 54 56 58 60 62 50 582 54 56 58 60 62 
MTM1 A2016 MTMM2 $2016 


Figure 2. Performance in final examinations vs. performance in mastery tests (MT) (NV = 295). 


Remembering that a larger proportion of MM1 students (than MM2) chose to sit the 
final examination, we suggest that this may have been a strategic choice, as the marks that 
students had earned for the mastery tests were so high that only a few more marks on the 
final were required to achieve a Credit. In MM2, however, the mastery test distribution 
was wider and more marks were required in the final examination for a Credit. It may be 
that students in MM2 made a strategic decision not to prepare for or to sit the final 
examination, if they decided not to aim for a grade higher than a Pass. We can also 
speculate that, as many students gained full marks on the mastery tests in MM1, they may 
have assumed that the final examination would be easy. From Table 1, we see that 79 
students sat the MMI final examination, but not the MM2 final examination. Looking at 
the mastery test scores of these students, we found that many of these students were using 
third attempts to pass the mastery tests and these were held towards the end of semester. It 


is possible that these students found that the difficulty of the mastery tests in MM2 
deterred them from attempting the final examination, either because they ran out of study 
time or because they felt the material would be too difficult. It must be remembered that 
these subjects have large enrolments and it is unlikely that the same motivations regarding 
the final examinations apply to all students. 

The second research question focuses on the perceived level of difficulty of the 
mastery tests. In the MMI SFS, students had the opportunity to respond to the questions 
“What did you particularly like in this subject?” and “Please suggest any improvements 
that could be made to this subject.” While there were many positive comments about the 
features of ML (that it reduced examination stress, kept students learning throughout the 
semester, and allowed students to re-sit to improve their marks), three students commented 
that they found the tests too easy, and each had something different to add to this: 


Also, the mastery quiz assessment is too easy and I don't think should be weighted more than the 
exam as it kinda gives of a feeling that the content in the mastery test is all we need to know. 


There should be assessment tasks that cover more difficult content instead of the mastery tests, as 
this will provide a better indication on how prepared we are for the final exam. 


Mastery tests were too easy, at least one or two harder questions in mastery would have pushed 
people to actually learn how to do difficult questions rather than just breezing though and not 
needing to work for the final exam. (All comments from MM1 SFS Autumn 2016) 


In the MM2 SFS, there were also appreciative comments about the way that ML 
provided motivation to study throughout the semester. Comments that the tests were too 
easy were rare; however, it was clear that some students were aware that others had found 
ways to pass with shallow learning and pattern matching: 

I feel like a lot of fellow students will not have benefitted in the way that they could have because 


they simply memorised all the variants of the questions in the mastery-tests. (MM2 SFS Spring 
2016) 


Towards the end of Autumn semester 2016, all students in MM1 were invited to 
complete an online survey, and this was an opportunity for staff to ask explicitly for 


feedback about various issues. There were 76 responses to the survey, many of which were 
incomplete. Table 2 shows the questions relevant to our research questions for this paper. 


Table 2 
Complete Responses to Optional Survey of MMI Autumn 2016 Students (N = 60): 
Questions about Mastery Learning 


Percentage Sub-group of 40 
Question agree or who sat both 
strongly agree final exams 


Doing the Mastery Test improved my confidence 


9 0 
with maths in MM1 as the semester progressed. we 88% 
I felt confident of doing well in the MM1 final 60%, su 
exam. 
I feel confident that I am ready to undertake MM2. 75% 78% 


As shown in Table 2, many students report gaining confidence with mathematics by 
doing the mastery tests. However, this does not include feeling confident about doing well 


in the final examination. It should be noted that the sub-group of 40 volunteers were 
predominantly high-achieving students, not representative of the class as a whole. 

What do the survey data reveal about the perceived level of difficulty of the mastery 
tests? Students were invited to respond in their own words to the question “What did you 
think about the mastery tests in MM1?” Answers were coded by two members of the 
research team working together. Codes were chosen to reflect the main opinions expressed. 
Regarding the ease of mastery tests, four said that they were “too easy” and 13 said “easy”. 
As indicated, 17 of the 40 replies said that the mastery tests were easy. This is reinforced 
by the fact that many students finished mastery tests well under the time allocated. 


Analysis for Students in MM1 Spring 2015 


The second cohort described in this paper consists of the 76 students who attempted 
MM1 in Spring 2015 and MM2 in Autumn 2016. This second group consists mainly of 
students who took the preliminary subject Foundation Mathematics in Autumn 2015. Table 
3 shows the numbers who chose to sit the final examinations in each subject. As for Table 
1, we see a drop in the participation in the optional final examination in MM2. 


Table 3 
Final Examination Participation of Students Who Attempted MM1 and MM2 Consecutively 
in the Spring 2015 and Autumn 2016 Semesters 


Did not sit MM2 exam Sat MM2 exam Total 
Did not sit MM1 exam 17 7 24 
Sat MM1 exam 14 38 52 
Totals 31 45 76 


From the data in Table 3, we see that 52/76 or 68% of these students sat the final 
examination in MM1 but only 45/76 or 59% chose to sit the examination in MM2. This is 
lower than the 89% and 72% respectively from the larger cohort (Autumn 2016 MM1 to 
Spring 2016 MM2). A possible reason is that many of these students arrive at university 
underprepared for mathematics and need all their study efforts just to pass the mastery 
tests. The final examination was not mentioned by students in the SFS for these subjects, 
while the majority of the comments about ML were positive. 
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Figure 3. Performance in mastery tests (MT) and in final examinations (N = 38). 


Figure 3 uses boxplots to show the scores of this cohort on the mastery tests and the 
final examinations. Comparing this with Figure 1, we conclude that this cohort found the 
mastery tests and the final examinations more difficult. There is a clear difference in the 
way that the MMI mastery test scores of students in the larger cohort (MM1 in Autumn 
2016, as shown in Figure 1) are skewed, with many finding the tests easy and scoring close 
to 100% on mastery tests. This may reflect the fact that many of the students in the larger 
cohort have studied more advanced levels of mathematics at school, and have an advantage 
for at least the first two mastery tests. In Figure 4, we note similar trends as in Figure 2. 
However, the MM1 mastery tests were not found by many students to be easy. 
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Figure 4. Performance in final examinations vs. performance in mastery tests (MT) (NV = 38). 


In the SFS for the smaller cohort, several students commented that they like the 
mastery test regime. For example: 


I really liked the mastery exam, each mastery exam tested our knowledge of a specific section of the 
math topics and thus helped us maximise our knowledge of each topic separately. This really helped 
as we had an overall understanding of each topic separately and then the final exam tested our 
knowledge as whole, very neat and knowledgeable setup. (MM1 SFS Spring 2015) 


It was heartening to read this perceptive comment: 


I absolutely loved the fact that we got several chances to attempt the mastery tests, which allowed 
me to learn from my mistakes and be allowed a chance to rectify them... Being allowed the proper 
opportunity to improve where I lacked is something I will take forward from this subject. Thank you 
for making maths fun. (MM2 SFS Autumn 2016) 


Discussion and Conclusions 


Regarding our research questions, we can say the following: 

1. A number of students from both cohorts chose not to sit the final examination. 
This could well indicate a lack of interest in higher grades and an unwillingness 
to engage with the more difficult material that was tested in the final 
examinations. The different shapes of the boxplots might also be an indicator. 
Further investigation is required. 

2. In the survey, 17 students said that the MM1 mastery tests were easy or too 
easy, while none said they were too difficult. In the SFS for both subjects, most 
comments about the mastery tests focussed on positive features and/or that they 
were easy. 


Implications for Future Practice 


A major concern for the teaching team is that many students are not fully engaging 
with opportunities to learn more complex problem solving and modelling. This material is 
in the tutorials (which are not well attended, especially towards the end of semester), and is 
tested in the final examinations. To encourage students to engage more with this material, 
the following changes are being considered for 2017: 

¢ Making the final examinations compulsory and worth more than the current 
37.5% of the overall mark, and making the achievement of some minimum 
mark in the final examination a requisite for a pass. 

¢ Ensuring that the mastery tests reward effort spent learning about problem- 
solving, and enlarging the sizes of pools of questions from which the questions 
are randomly drawn. 

¢ Offering students a weekly test to see if they are prepared for more challenging 
activities in “advanced” tutorials. These would be problem-based, and 
encourage collaborative efforts with students working in groups. 

¢ Encouraging students to better utilise existing student study support in the form 
of U:PASS (UTS Peer Assisted Study Success; University of Technology 
Sydney, 2017b) and the Maths and Science Study Support Centre activities 
including the drop-in room (University of Technology Sydney, 2017a). 

Our concerns about the nature of the online computer-based mastery tests are not new. 
Bradley (2016) points out that “It is possible for a student to repeat the problem enough 
times to learn the procedure for getting the correct answer without mastering the intended 
mathematical concept” (p. 28). Finding the right balance of activities and assessment 
rewards to encourage both conceptual and procedural learning is an ongoing task. 
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